Duplicate Web Pages Detection with the Support of 2d Table Approach

Duplicate and near duplicate web pages are stopping the process of search engine. As a consequence of duplicate and near duplicates, the common issue for the search engines is raising the indexed storage pages. This high storage memory will slow down the process which automatically increases the serving cost. Finally, the duplication will be raised while gathering the required data from the various sources based on the user’s query. The duplication will definitely slow down the information retrieval process. Duplication is nothing but the similar content or documents located under various sites. Content duplication can be taken place at different forms and levels such as exact document copy, paragraph copy, sentence copy, single word changes and sentence structure changes. Duplication detection is the process of identifying the multiple representations of a same real world object. In this paper, the content duplication is identified using two dimensional (2D) text matrix approach. By using the proposed 2D matrix approach, the system was able to detect duplicate web pages with a high precision value 92% is highlighting that the duplicate web page detection with the 2D technique is performing well Keyword: Near Duplicate Detection, 2D Approach, Information Retrieval, Content Duplicate.